Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added mirroring support #327

Merged
merged 34 commits into from
Nov 29, 2024
Merged

feat: added mirroring support #327

merged 34 commits into from
Nov 29, 2024

Conversation

Ak-sky
Copy link
Member

@Ak-sky Ak-sky commented Oct 29, 2024

Description

Added ES mirroring support for disaster recovery usecase.
Git Issue

Release required?

  • No release
  • Patch release (x.x.X)
  • Minor release (x.X.x)
  • Major release (X.x.x)
Release notes content
  • Added support for mirroring for disaster recovery usecase.

Run the pipeline

If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.

Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:

/run pipeline

Checklist for reviewers

  • If relevant, a test for the change is included or updated with this PR.
  • If relevant, documentation for the change is included or updated with this PR.

For mergers

  • Use a conventional commit message to set the release level. Follow the guidelines.
  • Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
  • Use the Squash and merge option.

@JunliWang
Copy link
Contributor

JunliWang commented Oct 29, 2024

mirroring needs a parameters_json with a structure like this, so that cluster provisioning will come with additional workerpool and mirrormaker application running on it.
source_crn, source_alias, and target_alias are user input and required.

parameters_json = jsonencode(
    {
      mirroring = {
        source_crn   = data.ibm_resource_instance.es_instance_source.id
        source_alias = "source-alias"
        target_alias = "target-alias"
        }
     }
 )

user also has the options to configure it to be more complex like below:

parameters_json = jsonencode(
   {
     mirroring = {
       source_crn   = data.ibm_resource_instance.es_instance_source.id
       source_alias = "source-alias"
       target_alias = "target-alias"
       options = {
         topic_name_transform = {
            type = "rename" 
            rename = {
              add_prefix = "add_prefix"
              add_suffix = "add_suffix"
              remove_prefix = "remove_prefix"
              remove_suffix = "remove_suffix"
            }
         }
         group_id_transform = {
           type = "rename" 
           rename = {
             add_prefix = "add_prefix"
             add_suffix = "add_suffix"
             remove_prefix = "remove_prefix"
             remove_suffix = "remove_suffix"
           }
         }
       }
     }
   }
 )

@kccox
Copy link
Contributor

kccox commented Oct 31, 2024

Note, you need plan = "enterprise-3nodes-2tb" to use mirroring, but provisioning of an enterprise instance can take two to three hours. You probably do not want to test that in pr_test.go.

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 5, 2024

Addressing changes with the latest stable provider release https://registry.terraform.io/providers/IBM-Cloud/ibm/1.71.0

@JunliWang
Copy link
Contributor

@Ak-sky
the resource ibm_event_streams_mirroring_config depends on the provider release v1.71.0, the parameters_json for mirroring does not, it is already supported by resource controller and Event Streams' service broker.

@JunliWang
Copy link
Contributor

example doc for mirroring
IBM-Cloud/terraform-provider-ibm#5767

@Ak-sky Ak-sky marked this pull request as ready for review November 18, 2024 14:16
@Ak-sky Ak-sky requested a review from akocbek as a code owner November 18, 2024 14:16
@Ak-sky Ak-sky changed the title [DONOTMERGE]feat: added mirroring support feat: added mirroring support Nov 18, 2024
@Ak-sky Ak-sky requested review from ocofaigh and JunliWang November 19, 2024 12:25
@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 19, 2024

@srikant-sahu, can you please have a look at this PR.

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 19, 2024

/run pipeline

Copy link
Member

@ocofaigh ocofaigh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mirror example will take too long to run in pr_test.go. I think we maybe add a permanent enterprise instance to our account, and update the fscloud example with mirror support. Then we can run the fscloud example in pr_test.go to test all enterprise features

examples/complete/variables.tf Outdated Show resolved Hide resolved
examples/complete/variables.tf Outdated Show resolved Hide resolved
main.tf Outdated Show resolved Hide resolved
@ocofaigh ocofaigh mentioned this pull request Nov 19, 2024
6 tasks
@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 22, 2024

/run pipeline

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 22, 2024

/run pipeline

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 24, 2024

/run pipeline

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 24, 2024

/run pipeline

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 24, 2024

@ocofaigh, PR pipeline is still failing due to permission error

╷
Error: [0m Error: [ERROR] Error creating resource key: You do not have the required permission to assign role 'Manager'. Contact the account owner to update your access. with resp code: {
│     "StatusCode": 403,

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 26, 2024

/run pipeline

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 26, 2024

@ocofaigh, PR pipeline is still failing due to permission error

╷
Error: [0m Error: [ERROR] Error creating resource key: You do not have the required permission to assign role 'Manager'. Contact the account owner to update your access. with resp code: {
│     "StatusCode": 403,

Error was coming due to the CBR rules.

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 26, 2024

/run pipeline

Copy link
Member

@ocofaigh ocofaigh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comments

variables.tf Outdated Show resolved Hide resolved
variables.tf Outdated Show resolved Hide resolved
variables.tf Outdated Show resolved Hide resolved
outputs.tf Outdated Show resolved Hide resolved
modules/fscloud/variables.tf Outdated Show resolved Hide resolved
variables.tf Outdated Show resolved Hide resolved
modules/fscloud/variables.tf Outdated Show resolved Hide resolved
main.tf Outdated Show resolved Hide resolved
examples/fscloud/variables.tf Show resolved Hide resolved
@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 29, 2024

/run pipeline

@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 29, 2024

/run pipeline

@Ak-sky Ak-sky requested a review from ocofaigh November 29, 2024 10:43
variables.tf Outdated Show resolved Hide resolved
@Ak-sky Ak-sky requested a review from ocofaigh November 29, 2024 12:28
@Ak-sky
Copy link
Member Author

Ak-sky commented Nov 29, 2024

/run pipeline

@ocofaigh ocofaigh merged commit 8503424 into main Nov 29, 2024
2 checks passed
@ocofaigh ocofaigh deleted the mirror branch November 29, 2024 16:17
@terraform-ibm-modules-ops
Copy link
Contributor

🎉 This PR is included in version 2.11.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants